onlin experi practic lesson pitfal avoid run control experi web control experi web survei practic guid knowledg discoveri data mine techniqu commonli find potenti pattern data fayyad al chapman al kdd applic involv post hoc analysi data limit identif correl recent semin work quasi experiment design jensen al attempt identifi causal relationship control experi standard techniqu multipl field random proper design experi establish causal scientif gold standard drug test softwar develop multipl techniqu defin product requir control experi provid assess impact featur custom behavior data mine case studi workshop call describ complet implement relat data mine year built experiment platform system exp microsoft capabl run analyz control experi web site servic goal acceler innov trustworthi experiment enabl scientif approach plan priorit featur design folei ran experi dozen microsoft properti tackl technic cultur challeng previous survei literatur control experi share technic challeng kohavi al paper focus problem commonli address technic paper cultur challeng lesson roi run control experi practic guid control experi web listen custom hippo onlin experi lesson learn focu mine beacon lesson challeng world dcommerc proceed tenth acm sigkdd intern confer knowledg discoveri data mine visual rfm segment lesson challeng mine retail dcommerc data abstract architectur blue martini softwar commerc suit support data collect transform data mine incept clickstream collect applic server layer high level event log data automat transform data warehous meta data common problem plagu data mine weblog session conflat multi sourc data obviat allow concentr actual data mine goal briefli review architectur discuss lesson learn year challeng face lesson challeng present dimens busi level technic data mine lifecycl stage data collect data warehous construct busi intellig deploy lesson challeng wide applic data mine domain retail commerc keyword data mine data analysi busi intellig web analyt web mine olap webkdd mine web log data custom touch point emerg trend busi analyt paper discuss technolog enterpris adopt trend area busi analyt kei consum analyt busi user person job directli relat analyt se merchandis market salesperson typic analyt tool improv result busi process dimens profit time market fortun data mine analyt applic busi intellig system integr transact system creat close loop oper analys data analyz faster analysi result quickli reflect busi action mine inform deploi broader busi audienc take advantag busi analyt everydai activ analyt regularli multipl area includ sale market suppli chain optim fraud detect web mine integr dcommerc data mine architectur challeng show commerc domain provid ingredi success data mine describ integr architectur support integr architectur dramat reduc pre process clean data understand effort document time knowledg discoveri project emphas data collect applic server layer web server order support log data metadata essenti discoveri process describ data transform bridg requir transact process system custom event stream clickstream data warehous detail mine workbench provid multipl view data report data mine algorithm visual olap conclud set challeng real world perform associ rule algorithm studi compar associ rule algorithm real world dataset artifici dataset experiment result confirm perform improv previous claim author artifici data gain carri real dataset indic overfit algorithm ibm artifici dataset importantli found choic algorithm matter support level gener rule practic support level gener rule human handl suffici predict purpos data load ram apriori finish process minut dataset observ super exponenti growth number rule dataset chang support increas number rule million billion impli narrow rang support valu choic algorithm irrelev categori subject descriptor databas manag applic data mine keyword data mine associ rule benchmark comparison frequent itemset market basket analysi affin analysi mine dcommerc data good bad ugli organ conduct electron commerc commerc greatli benefit insight data mine transact clickstream data insight help improv electron channel web site learn vehicl bigger organ conduct busi brick mortar store commerc site serv earli alert system emerg pattern laboratori experiment success data mine ingredi need commerc good web server log commonli sourc data mine commerc data design debug web server data provid insuffici requir heurist reconstruct event event log web server log limit sourc data mine bad problem deal web server log data resolv properli architect ecommerc site gener data need mine good architectur challeng problem remain hard solv ugli lesson metric base mine real commerc data present categori subject descriptor databas manag applic data mine artifici intellig learn gener term measur design experiment human factor keyword commerc data mine applic server web server web site architectur applic data mine electron commerc data mine project substanti hurdl heroic eort block success complet project ve desiderata success seldom present data mine applic data rich descript wide custom record potenti eld data mine algorithm search obviou correl larg volum data larg model space rich data demand train instanc build reliabl model control reliabl data collect manual data entri integr legaci system notori problemat fulli autom collect consider abil evalu result substanti demonstr return invest convinc ron kohavi foster provost eas integr exist process pilot studi mine dcommerc data good bad ugli organ conduct electron commerc commerc greatli benefit insight data mine transact clickstream data insight help improv electron channel web site learn vehicl bigger organ conduct busi brick mortar store commerc site serv earli alert system emerg pattern laboratori experiment success data mine ingredi need commerc good web server log commonli sourc data mine commerc data design debug web server data provid insuffici requir heurist reconstruct event event log web server log limit sourc data mine bad problem deal web server log data resolv properli architect ecommerc site gener data need mine good architectur challeng problem remain hard solv ugli lesson metric base mine real commerc data present categori subject descriptor databas manag applic data mine artifici intellig learn gener term measur design experiment human factor keyword commerc data mine applic server web server web site architectur applic data mine electron commerc data mine project substanti hurdl heroic eort block success complet project ve desiderata success seldom present data mine applic data rich descript wide custom record potenti eld data mine algorithm search obviou correl larg volum data larg model space rich data demand train instanc build reliabl model control reliabl data collect manual data entri integr legaci system notori problemat fulli autom collect consider abil evalu result substanti demonstr return invest convinc ron kohavi foster provost eas integr exist process pilot studi web mine dcommerc workshop session titl integr dcommerc data mine architectur challeng show commerc domain provid ingredi success data mine describ integr architectur support integr architectur dramat reduc pre process clean data understand effort document time knowledg discoveri project emphas data collect applic server layer web server order support log data metadata essenti discoveri process describ data transform bridg requir transact process system custom event stream clickstream data warehous detail mine workbench provid multipl view data report data mine algorithm visual olap conclud set challeng applic data mine electron commerc data mine project substanti hurdl heroic eort block success complet project ve desiderata success seldom present data mine applic data rich descript wide custom record potenti eld data mine algorithm search obviou correl larg volum data larg model space rich data demand train instanc build reliabl model control reliabl data collect manual data entri integr legaci system notori problemat fulli autom collect consider abil evalu result substanti demonstr return invest convinc ron kohavi foster provost eas integr exist process pilot studi integr data mine vertic solut webkdd web mine dcommerc kdd dcup organ report peel onion describ kdd cup yearli competit data mine time cup includ insight problem addit predict problem pose challeng knowledg discoveri evalu criteria highlight peel onion drill deeper reason initi pattern found chronicl data gener phase start collect site convers star schema warehous data cleans data obfusc privaci protect data aggreg describ inform particip includ question site structur market calendar data schema final discuss interest insight common mistak lesson learn winner announc describ experi lesson page paper empir comparison vote classif algorithm bag method vote classif algorithm bag adaboost shown success improv accuraci classifi artifici real world dataset review algorithm describ larg empir studi compar variant conjunct decis tree induc variant naiv bay induc purpos studi improv understand algorithm perturb reweight combin techniqu affect classif error provid bia varianc decomposit error show method variant influenc term allow determin bag reduc varianc unstabl method boost method adaboost arc reduc bia varianc unstabl method increas varianc naiv bay stabl observ arc behav differ adaboost reweight resampl indic fundament differ vote variant introduc paper includ prune versu prune probabilist estim weight perturb wag backfit data found bag improv probabilist estim conjunct prune data backfit measur tree size show interest posit correl increas averag tree size adaboost trial success reduc error compar squar error vote method vote method show vote method lead larg signific reduct squar error practic problem aris implement boost algorithm explor includ numer instabl underflow scatterplot graphic show adaboost reweight instanc emphas hard area outlier nois prune decis tree misclassif cost describ experiment studi prune method decis tree classifi goal minim loss error addit common method error minim cart cost complex prune error base prune studi extens cost complex prune loss prune variant base laplac correct perform empir comparison method evalu respect loss found appli laplac correct estim probabl distribut leav benefici prune method unlik error minim surprisingli perform prune led result par method term evalu criteria main advantag prune reduct decis tree size factor ten method domin dataset domain prune mechan case accuraci estim compar induct algorithm analyz critic classif accuraci compar classifi natur data set provid investig roc analysi standard machin learn algorithm standard benchmark data set result rais concern accuraci compar classifi drawinto question conclus drawn studi present describ demonstr proper roc analysi compar studi machin learn research argu methodolog prefer make practic choic draw scientif conclus target busi user decis tabl classifi busi user analyst commonli spreadsheet plot analyz understand data line analyt process olap user ad flexibl pivot data di erent attribut drill multi dimension cube aggreg machin learn research concentr hypothesi space foreign user hyperplan perceptron neural network bayesian network decis tree nearest neighbor paper advoc decis tabl classifi easi line busi user understand describ variant algorithm learn decis tabl compar perform describ visual mechan implement mineset perform decis tabl compar algorithm result classifi fewer attribut comprehens introduct abo guest editor introduct appli research machin learn option decis tree major vote describ experiment studi option decis tree major vote option decis tree gener regular decis tree allow option node addit decis node node test conduct commonli singl test goal explor option node control growth tree addit complex util limit option decis tree reduc error decis tree real world problem combin multipl option motiv similar vote algorithm learn multipl model combin predict unlikevot algorithm option decis tree singl structur classifi decis tree interpret easili human result show test problem achiev signific reduct error rate tree restrict level mineset integr system data mine mineset tm silicon graphic interact system data mine integr power technolog databas access analyt data mine data visual support knowledg discoveri process data access prepar iter analysi visual deploy mineset base client server architectur scale larg databas databas access compon rich set oper preprocess transform store data form visual analyt mine visual capabl direct data visual exploratori analysi includ tool displai high dimension data geograph hierarch inform analyt mine algorithm identifi potenti interest model data view visual tool special learn model parti vendor interfac mineset wrapper featur subset select featur subset select problem learn algorithm face problem select relev subset featur focu attent ignor rest achiev perform learn algorithm domain featur subset select method algorithm train data interact explor relat optim featur subset select relev wrapper method search optim featur subset tailor algorithm domain studi strength weak wrapper approach show improv origin design compar wrapper approach induct featur subset select relief filter base approach featur subset select signific improv accuraci real problem achiev famili induct algorithm decis tree naiv bay data mine mlc machin learn librari data mine algorithmsinclud machin learn statist analysi pattern recognit techniqu greatli improv understand data warehous widespread paper focu classif algorithm review multipl classif algorithm describ system call mlc design choos classif algorithm dataset make easi compar util algorithm specif dataset interest mlc workbench comparison librari class aid develop algorithm hybrid algorithm multi strategi algorithm algorithm gener hard code scratch discuss design issu interfac program visual result classifi introduct data warehous massiv amount data lazi decis tree lazi learn algorithm exemplifi nearestneighbor algorithm induc concis hypothesi train set induct process delai test instanc algorithm construct decis tree id cart creat singl decis tree train phase tree classifi test instanc test node construct tree good averag test classifi specif instanc propos lazi decis tree algorithm lazydt conceptu construct decis tree test instanc practic path construct cach scheme make algorithm fast algorithm robust respect miss valu resort complic method induct decis tree experi real artifici problem present introduct delai prefer error bia varianc decomposit loss function present bia varianc decomposit expect misclassif rate commonli loss function supervis classif learn bia varianc decomposit quadrat loss function serv import tool analyz learn algorithm decomposit offer commonli misclassif loss function recent work kong amp dietterich breiman decomposit suffer major shortcom potenti neg varianc decomposit avoid show practic naiv frequenc base estim decomposit term bias show correct bia illustr decomposit algorithm dataset uci repositori data mine mlc machin learn librari data mine algorithmsinclud machin learn statist analysi pattern recognit techniqu greatli improv understand data warehous widespread paper focu classif algorithm review multipl classif algorithm describ system call mlc design choos classif algorithm dataset make easi compar util algorithm specif dataset interest mlc workbench comparison librari class aid develop algorithm hybrid algorithm multi strategi algorithm algorithm gener hard code scratch discuss design issu interfac program visual result classifi introduct data warehous massiv amount data error dbase entropi dbase discret continu featur present comparison error base entropybas method discret continu featur studi includ extens empir comparison analysi scenario error minim inappropri discret criterion present discret method base decis tree algorithm compar exist entropi base discret algorithm emploi minimum descript length principl recent propos error base techniqu evalu discret method respect naiv bayesian classifi dataset uci repositori analyz comput complex method result entropi base mdl heurist outperform error minim averag analyz shortcom error base approach comparison entropi base method introduct real world classif data mine task involv con scale accuraci naiv dbay classifi decis dtree hybrid naiv bay induct algorithm previous shown surprisingli accur classif task condit independ assumpt base violat studi small databas show larger databas accuraci naiv bay scale decis tree propos algorithm nbtree induc hybrid decis tree classifi naiv bay classifi decis tree node univari split regular decis tree leav naiv bayesian classifi approach retain interpret naiv bay decis tree result classifi frequent outperform constitu larger databas test power decis tabl evalu power decis tabl hypothesi space supervis learn algorithm decis tabl simplest hypothesi space easi understand experiment result show artifici real world domain discret featur idtm algorithm induc decis tabl outperform state art algorithm surprisingli perform good dataset continu featur indic dataset machin learn requir featur featur valu describ increment method perform crossvalid applic increment learn algorithm includ idtm increment cross valid cross valid dataset idtm time linear number instanc number featur number label valu time incr supervis unsupervis discret continu featur supervis machin learn algorithm requir discret featur space paper review previou work continu featur discret identifi defin characterist method conduct empir evalu method compar bin unsupervis discret method entropi base puriti base method supervis algorithm found perform naiv bay algorithm significantli improv featur discret entropi base method fact test dataset discret version naiv bay slightli outperform averag show case perform induct algorithm significantli improv featur discret advanc experi perform significantli degrad interest phenomenon fact capabl local discret featur autmat paramet select minim estim error oblivi decis tree graph top ddown prune describ supervis learn algorithm eodg mutual inform build oblivi decis tree tree convert oblivi read decis graph oodg merg node level tree domain decis tree oodg perform approxim number node oodg smaller merg phase convert oblivi decis tree oodg deal replic problem prune mechan work top start root prune mechan suit find symmetri aid recov split irrelev featur happen tree construct introduct decis tree provid hypothesi space supervis machin learn algorithm suit dataset encount real world breiman friedman olshen amp stone quinlan studi cross dvalid bootstrap accuraci estim model select review accuraci estim method compar common method cross valid bootstrap recent experiment result artifici data theoret result restrict set shown select good classifi set classifi model select ten fold cross valid expens leaveon cross valid report largescal experi half million run naiv bay algorithm estim effect paramet algorithm real world dataset cross valid vari number fold fold stratifi bootstrap vari number bootstrap sampl result real word dataset similar method model select ten fold stratifi cross valid comput power fold bottom dup induct oblivi read donc decis graph strength limit featur subset select wrapper method overfit dynam search space topolog wrapper approach featur subset select search optim set featur made induct algorithm blackbox estim futur perform algorithm heurist guid search statist method featur subset select includ forward select backward elimin stepwis variant view simpl hill climb techniqu space featur subset util searchtofindagood featur subset discuss overfit problem search featur subset introduc compound oper dynam chang topolog search space util inform evalu featur subset show compound oper unifi previou approach deal relev irrelev featur improv featur subset select yield signific improv real world dataset bottom dup induct oblivi read donc decis graph strength limit bottom dup induct oblivi read donc decis graph investig oblivi read decis graph structur repres concept discret domain present bottom hill climb algorithm infer structur label instanc algorithm robust respect irrelev attribut experiment result show perform problem consid difficult symbol induct method monk problem pariti introduct top induct decis tree princip induct method symbol supervis learn tree structur repres hypothes target concept suffer wellknown problem notabl replic problem fragment problem replic problem forc duplic subtre disjunct concept fragment problem partit data fragment high ariti attrib bottom dup induct oblivi read donc decis graph investig oblivi read decis graph structur repres concept discret domain present bottom hill climb algorithm infer structur label instanc algorithm robust respect irrelev attribut experiment result show perform problem consid difficult symbol induct method monk problem pariti introduct top induct decis tree princip induct method symbol supervis learn tree structur repres hypothes target concept suffer wellknown problem notabl replic problem fragment problem replic problem forc duplic subtre disjunct concept fragment problem partit data fragment high ariti attrib irrelev featur subset select problem address problem find subset featur supervis induct algorithm induc small high accuraci concept examin notion relev irrelev show definit machin learn literatur adequ partit featur categori relev present definit irrelev degre relev definit improv understand behavior previou subset select algorithm defin subset featur sought featur select depend featur target concept induct algorithm describ method featur subset select cross valid applic induct algorithm discuss experi conduct id artifici real dataset mlc machin learn librari present mlc librari class tool supervis machin learn mlc gener learn algorithm end user main object provid research expert wide varieti tool acceler algorithm develop increas softwar reliabl provid comparison tool displai inform visual collect exist algorithm mlc attempt extract common algorithm decompos unifi view simpl coher extens paper discuss problem mlc aim solv design mlc current function research note decis list 